Analysis of results in Dependability Benchmarking: Can we do better?"

©2013 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Dependability benchmarking has become through the years more and more important in the process of systems evaluation. The increasing need for making systems more dependable in presence of perturbations has contributed to this fact. Nevertheless, even though many studies have focused on different areas related to dependability benchmarking, and some others have focused on the need of providing these benchmarks with good quality measures, there is still a gap in the process of the analysis of results. This paper focuses on providing a first glance at different approaches that may help filling this gap by making explicit the criteria followed in the decision making process.This work is partially supported by the Spanish project ARENES (TIN2012-38308-C02-01), the ANR French project AMORES (ANR-11-INSE-010), and the Intel Doctoral Student Honour Programme 2012.Martínez, M.; Andrés, DD.; Ruiz García, JC.; Friginal López, J. (2013). Analysis of results in Dependability Benchmarking: Can we do better?". IEEE. https://doi.org/10.1109/IWMN.2013.6663790

From measures to conclusions using Analytic Hierarchy Process in dependability benchmarkind

© 2014 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Dependability benchmarks are aimed at comparing and selecting alternatives in application domains where faulty conditions are present. However, despite its importance and intrinsic complexity, a rigorous decision process has not been defined yet. As a result, benchmark conclusions may vary from one evaluator to another, and often, that process is vague and hard to follow, or even nonexistent. This situation affects the repeatability and reproducibility of that analysis process, making difficult the cross-comparison of results between works. To mitigate these problems, this paper proposes the integration of the analytic hierarchy process (AHP), a widely used multicriteria decision-making technique, within dependability benchmarks. In addition, an assisted pairwise comparison approach is proposed to automate those aspects of AHP that rely on judgmental comparisons, thus granting consistent, repeatable, and reproducible conclusions. Results from a dependability benchmark for wireless sensor networks are used to illustrate and validate the proposed approach.This work was supported in part by the Spanish Project ARENES under Grant TIN2012-38308-C02-01 and in part by the Programa de Ayudas de Investigacion y Desarrollo through the Universitat Politecnica de Valencia, Valencia, Spain. The Associate Editor coordinating the review process was Dr. Dario Petri.Martínez Raga, M.; Andrés Martínez, DD.; Ruiz García, JC.; Friginal López, J. (2014). From measures to conclusions using Analytic Hierarchy Process in dependability benchmarkind. IEEE Transactions on Instrumentation and Measurement. 63(11):2548-2556. https://doi.org/10.1109/TIM.2014.2348632S25482556631

Ambient noise in wireless mesh networks: Evaluation and proposal of an adaptive algorithm to mitigate link removal

[EN] Ambient noise is one of the major problems in Wireless Mesh Networks (WMNs). It is responsible for adverse effects on communications such as packet dropping, which dramatically affects the behaviour of ad hoc routing protocols ,a key element of these networks. This issue is of prime importance for WMNs since the loss of communication links experienced by nodes may strongly increase the convergence time of the network. Furthermore, the dynamic nature of this problem makes it difficult to address with traditional techniques. The contribution of this paper goes in the direction of (i) exploring this problem by assessing the behaviour of three state-of-the-art routing protocols in the presence of ambient noise (OLSR, B.A.T.M.A.N and Babel) and (ii) improving the resilience capabilities of these protocols against ambient noise by proposing an algorithm for the link quality-based adaptive replication of packets, named LARK. The goal of LARK is to avoid the loss of communication links in the presence of high levels of ambient noise. The effectiveness of the proposal is experimentally assessed, thus establishing a new method to reduce the impact of ambient noise on WMNs.This work is partially supported by the Spanish projects ARENES (TIN2012-38308-C02-01) and SEMSECAP (TIN-2009-13825), the ANR French project AMORES (ANR-11-INSE-010), and the Intel Doctoral Student Honour Programme 2012.Friginal López, J.; Ruiz García, JC.; Andrés Martínez, DD.; Bustos Rodríguez, A. (2014). Ambient noise in wireless mesh networks: Evaluation and proposal of an adaptive algorithm to mitigate link removal. Journal of Network and Computer Applications. 41:505-516. https://doi.org/10.1016/j.jnca.2014.02.004S5055164

Multi-criteria analysis of measures in benchmarking: Dependability benchmarking as a case study

This is the author’s version of a work that was accepted for publication in The Journal of Systems and Software. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Multi-criteria analysis of measures in benchmarking: Dependability benchmarking as a case study. Journal of Systems and Software, 111, 2016. DOI 10.1016/j.jss.2015.08.052.Benchmarks enable the comparison of computer-based systems attending to a variable set of criteria, such as dependability, security, performance, cost and/or power consumption. It is not despite its difficulty, but rather its mathematical accuracy that multi-criteria analysis of results remains today a subjective process rarely addressed in an explicit way in existing benchmarks. It is thus not surprising that industrial benchmarks only rely on the use of a reduced set of easy-to-understand measures, specially when considering complex systems. This is a way to keep the process of result interpretation straightforward, unambiguous and accurate. However, it limits at the same time the richness and depth of the analysis process. As a result, the academia prefers to characterize complex systems with a wider set of measures. Marrying the requirements of industry and academia in a single proposal remains a challenge today. This paper addresses this question by reducing the uncertainty of the analysis process using quality (score-based) models. At measure definition time, these models make explicit (i) which are the requirements imposed to each type of measure, that may vary from one context of use to another, and (ii) which is the type, and intensity, of the relation between considered measures. At measure analysis time, they provide a consistent, straightforward and unambiguous method to interpret resulting measures. The methodology and its practical use are illustrated through three different case studies from the dependability benchmarking domain, a domain where various different criteria, including both performance and dependability, are typically considered during analysis of benchmark results.. Although the proposed approach is limited to dependability benchmarks in this document, its usefulness for any type of benchmark seems quite evident attending to the general formulation of the provided solution. © 2015 Elsevier Inc. All rights reserved.This work is partially supported by the Spanish project ARENES (TIN2012-38308-C02-01), ANR French project AMORES (ANR-11-INSE-010), the Intel Doctoral Student Honour Programme 2012, and the "Programa de Ayudas de Investigacion y Desarrollo" (PAID) from the Universitat Politecnica de Valencia.Friginal López, J.; Martínez, M.; De Andrés, D.; Ruiz, J. (2016). Multi-criteria analysis of measures in benchmarking: Dependability benchmarking as a case study. Journal of Systems and Software. 111:105-118. https://doi.org/10.1016/j.jss.2015.08.052S10511811

A survey of evaluation platforms for ad hoc routing protocols: a resilience perspective

Routing protocols allow for the spontaneous formation of wireless multi-hop networks without dedicated infrastructure, also known as ad hoc networks. Despite significant technological advances, difficulties associated with the evaluation of ad hoc routing protocols under realistic conditions, still hamper their maturation and significant roll out in real world deployments. In particular, the resilience evaluation of ad hoc routing protocols is essential to determine their ability of keeping the routing service working despite the presence of changes, such as accidental faults or malicious ones (attacks). However, the resilience dimension is not always addressed by the evaluation platforms that are in charge of assessing these routing protocols. In this paper, we provide a survey covering current state-of-the-art evaluation platforms in the domain of ad hoc routing protocols paying special attention to the resilience dimension. The goal is threefold. First, we identify the most representative evaluation platforms and the routing protocols they have evaluated. Then, we analyse the experimental methodologies followed by such evaluation platforms. Finally, we create a taxonomy to characterise experimental properties of such evaluation platforms.This work is partially supported by the Spanish Project ARENES (TIN2012-38308-C02-01), the ANR French Project AMORES (ANR-11-INSE-010), and the Intel Doctoral Student Honour Programme 2012.Friginal López, J.; Andrés Martínez, DD.; Ruiz García, JC.; Martínez Raga, M. (2014). A survey of evaluation platforms for ad hoc routing protocols: a resilience perspective. Computer Networks. 75(A):395-413. https://doi.org/10.1016/j.comnet.2014.09.010S39541375

An experimental methodology to evaluate the resilience of ad hoc routing protocols

Friginal López, J. (2013). An experimental methodology to evaluate the resilience of ad hoc routing protocols [Tesis doctoral]. Editorial Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/1848

Evaluating the behaviour of real ad hoc networks through attack injection

Mobile ad hoc networks are distributed systems made of wireless mobile nodes that can freely and dynamically self-organize into arbitrary and temporary topologies. They allow people (and devices) to seamlessly interconnect in areas with no pre-existing communication infrastructure. Nowadays trustworthiness has become essential for the practical exploitation and use of ad hoc networks. This research ambitions the definition of a experimental approach for deriving performance and dependability measures from real wireless ad hoc networks.Friginal López, J. (2009). Evaluating the behaviour of real ad hoc networks through attack injection. http://hdl.handle.net/10251/14359Archivo delegad

Un estudio sobre la seguridad en los sistemas de información

Friginal López, J. (2011). Un estudio sobre la seguridad en los sistemas de información. http://hdl.handle.net/10251/11170.Archivo delegad

AMORES L1.2 - A Privacy Risk Assessment Methodology for Location-Based Systems

The proposed process is based on classical risk assessment process as defined in the risk manage ment process. We adapt the different steps to the domain of geo-privacy, and we propose a meta-model of concepts used for this study. This meta-model has been successfully used to compute privacy risk and its relation with attack trees. The feasibility of each step of our methodology is illustrated through the case study of dynamic carpooling of the AMORES project (see report L.1.1.a)

Towards privacy-driven design of a dynamic carpooling system

International audienceDynamic carpooling (also known as instant or ad-hoc ridesharing) is a service that arranges one-time shared rides on very short notice. This type of carpooling generally makes use of three recent technological advances: (i) Navigation devices to determine a driver's route and arrange the shared ride; (ii) smartphones for a traveller to request a ride from wherever she happens to be; and (iii) social networks to establish trust between drivers and passengers. However, the mobiquitous environment in which dynamic carpooling is expected to operate, raises several privacy issues. Among all the personal identifiable information, learning the location of an individual is one of the greatest threats against her privacy. For instance, the spatio-temporal data of an individual can be used to infer the location of her home and workplace, to trace her movements and habits, to learn information about her centre of interests or even to detect a change from her usual behaviour. Therefore, preserving location privacy is a major issue to be able to leverage the possibilities offered by dynamic carpooling. In this paper we use the principles of privacy-by-design to integrate the privacy aspect in the design of dynamic carpooling, henceforth increasing its public (and political) acceptability and trust